Data Description:¶
The actual concrete compressive strength (MPa) for a given mixture under a specific age (days) was determined from laboratory. Data is in raw form (not scaled). The data has 8 quantitative input variables, and 1 quantitative output variable, and 1030 instances (observations).
Domain:¶
Cement manufacturing
Context:¶
Concrete is the most important material in civil engineering. The concrete compressive strength is a highly nonlinear function of age and ingredients. These ingredients include cement, blast furnace slag, fly ash, water, superplasticizer, coarse aggregate, and fine aggregate.
Attribute Information:¶
- Cement: measured in kg in a m3 mixture
- Blast: measured in kg in a m3 mixture
- Fly ash : measured in kg in a m3 mixture
- Water : measured in kg in a m3 mixture
- Superplasticizer : measured in kg in a m3 mixture
- Coarse Aggregate : measured in kg in a m3 mixture
- Fine Aggregate : measured in kg in a m3 mixture
- Age: day (1~365)
- Concrete compressive strength :measured in MPa
Objective:¶
Modeling of strength of high performance concrete using Machine Learning
Importing Relevant Libraries¶
In [376]:
#for data manipulation
import pandas as pd
import numpy as np
#for ploting
import matplotlib.pyplot as plt
%matplotlib inline
import seaborn as sns
sns.set(color_codes=True)
#data tranformation and feature generation
from scipy.stats import zscore,norm
from sklearn.preprocessing import PolynomialFeatures
from sklearn.decomposition import PCA
from scipy.stats import pearsonr
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import MinMaxScaler,StandardScaler
from sklearn.pipeline import make_pipeline
from sklearn.linear_model import LinearRegression,Ridge,Lasso
from sklearn.svm import SVR
from sklearn.ensemble import (RandomForestRegressor, GradientBoostingRegressor,AdaBoostRegressor,BaggingRegressor)
from sklearn.preprocessing import PolynomialFeatures
import warnings
warnings.filterwarnings('ignore')
from sklearn.cluster import KMeans
#importing the K fold
from sklearn.model_selection import KFold
#importing the cross validation score
from sklearn.model_selection import cross_val_score
from sklearn.model_selection import RepeatedStratifiedKFold
from sklearn.model_selection import GridSearchCV
from sklearn.utils import resample
from matplotlib import pyplot
Reading the dataset¶
In [2]:
#reading the data
data=pd.read_csv("concrete.csv")
In [3]:
data.head(10)
Out[3]:
- From above table we found that there are 8 independent variable and one dependent variable
- All the records are numeric
Univariate Analysis¶
In [4]:
data.dtypes #to find the data types of each attributes
Out[4]:
In [5]:
data.shape #no of rows and columns in the dataframe
Out[5]:
There are 1030 rows and 9 columns in the given dataset
Checking the presence of missing values¶
In [6]:
data.isnull().sum()
Out[6]:
There are no missing values in the given dataset
Descriptive statistics of each & every column¶
In [7]:
data.describe().transpose()
Out[7]:
Five Point Summary¶
In [8]:
summary=data.describe().transpose()
summary[['min','25%','50%','75%','max']]
Out[8]:
In [9]:
data.skew(numeric_only = True)
Out[9]:
Skewness with positive values indicates data is skewed towards right. Skewness with negative values indicates data is skewed towards left
In [10]:
# A quick check to find columns that contain outliers
fig=plt.figure(figsize=(15,7))
ax=sns.boxplot(data=data,orient='v')
From the above graph we found that the attributes other than cement,ash and coarseagg are having outliers
Distribution of Independent attributes¶
cement¶
In [11]:
plt.figure(figsize=(20,20))
#boxplot
plt.subplot(3,3,1)
sns.boxplot(data.cement,showfliers=True,color="c").set_title("Distribution Of Cement")
# distplot
ax=plt.subplot(3,3,2)
sns.distplot(data.cement,color='m').set_title("cement Vs Frequency")
ax.axvline(data.cement.mean(),color='r',linestyle='--',label='Mean',linewidth=1.2)
ax.axvline(data.cement.median(),color='g',linestyle='--',label='Median',linewidth=1.2)
ax.legend(loc='best')
#histogram plot
plt.subplot(3,3,3)
data.cement.plot.hist(color='g').set_title("cement Vs Frequency");
Observation¶
- The column is almost Normally Distributed
- Mean and Median are almost the same
- Most of the values are between 192 to 350
- There are no outliers
slag¶
In [12]:
plt.figure(figsize=(20,20))
#boxplot
plt.subplot(3,3,1)
sns.boxplot(data.slag,showfliers=True,color='c').set_title("Distribution of Slag")
#dist plot
ax=plt.subplot(3,3,2)
sns.distplot(data.slag,color='m').set_title("slag Vs Frequency")
ax.axvline(data.slag.mean(),color='r',linestyle='--',label='Mean',linewidth=1.2)
ax.axvline(data.slag.median(),color='g',linestyle='--',label='Median',linewidth=1.2)
ax.legend(loc='best')
#histogram plot
plt.subplot(3,3,3)
data.slag.plot.hist(color='g').set_title("slag Vs Frequency");
Observation¶
- The column is skewed towards right
- Mean and Median are not the same
- 25% of the values are 0
- Most of the values are between 0 to 142
- There are outliers
- It has Three Gaussians
Ash¶
In [13]:
plt.figure(figsize=(20,20))
#boxplot
plt.subplot(3,3,1)
sns.boxplot(data.ash,showfliers=True,color='c').set_title("Distribution of Ash")
#dist plot
ax=plt.subplot(3,3,2)
sns.distplot(data.ash,color='m').set_title("ash Vs Frequency")
ax.axvline(data.ash.mean(),color='r',linestyle='--',label='Mean',linewidth=1.2)
ax.axvline(data.ash.median(),color='g',linestyle='--',label='Median',linewidth=1.2)
ax.legend(loc='best')
#histogram plot
plt.subplot(3,3,3)
data.ash.plot.hist(color='g').set_title("ash Vs Frequency");
Observation¶
- The column is skewed towards right
- 50% of the values are 0
- Mean and Median are not the same
- Most of the values are between 0 to 118
- There are no outliers
- It has Two Gaussians
water¶
In [14]:
plt.figure(figsize=(20,20))
#boxplot
plt.subplot(3,3,1)
sns.boxplot(data.water,showfliers=True,color='c').set_title("Distribution of water")
#dist plot
ax=plt.subplot(3,3,2)
sns.distplot(data.water,color='m').set_title("water Vs Frequency")
ax.axvline(data.water.mean(),color='r',linestyle='--',label='Mean',linewidth=1.2)
ax.axvline(data.water.median(),color='g',linestyle='--',label='Median',linewidth=1.2)
ax.legend(loc='best')
#histogram plot
plt.subplot(3,3,3)
data.water.plot.hist(color='g').set_title("water Vs Frequency");
Observation¶
- The column is slightly skewed towards left
- Mean and Median are not the same
- Most of the values are between 164 to 192
- There are outliers
- It has Three Gaussians
superplastic¶
In [15]:
plt.figure(figsize=(20,20))
#boxplot
plt.subplot(3,3,1)
sns.boxplot(data.superplastic,showfliers=True,color='c').set_title("Distribution of superplastic")
#dist plot
ax=plt.subplot(3,3,2)
sns.distplot(data.superplastic,color='m').set_title("superplastic Vs Frequency")
ax.axvline(data.superplastic.mean(),color='r',linestyle='--',label='Mean',linewidth=1.2)
ax.axvline(data.superplastic.median(),color='g',linestyle='--',label='Median',linewidth=1.2)
ax.legend(loc='best')
#histogram plot
plt.subplot(3,3,3)
data.superplastic.plot.hist(color='g').set_title("superplastic Vs Frequency");
Observation¶
- The column is skewed towards right
- Mean and Median are almost the same
- 25% of the values are 0
- Most of the values are between 0 to 10
- There are outliers
- It has Two Gaussians
coarseagg¶
In [16]:
plt.figure(figsize=(20,20))
#boxplot
plt.subplot(3,3,1)
sns.boxplot(data.coarseagg,showfliers=True,color='c').set_title("Distribution of coarseagg")
#dist plot
ax=plt.subplot(3,3,2)
sns.distplot(data.coarseagg,color='m').set_title("coarseagg Vs Frequency")
ax.axvline(data.coarseagg.mean(),color='r',linestyle='--',label='Mean',linewidth=1.2)
ax.axvline(data.coarseagg.median(),color='g',linestyle='--',label='Median',linewidth=1.2)
ax.legend(loc='best')
#histogram plot
plt.subplot(3,3,3)
data.coarseagg.plot.hist(color='g').set_title("coarseagg Vs Frequency");
In [ ]:
Observation¶
- The column is almost Normally Distributed
- Mean and Median are almost the same
- Most of the values are between 932 to 1029
- There are no outliers
- It has Three Gaussians
fineagg¶
In [17]:
plt.figure(figsize=(20,20))
#boxplot
plt.subplot(3,3,1)
sns.boxplot(data.fineagg,showfliers=True,color='c').set_title("Distribution of fineagg")
#dist plot
ax=plt.subplot(3,3,2)
sns.distplot(data.fineagg,color='m').set_title("fineagg Vs Frequency")
ax.axvline(data.fineagg.mean(),color='r',linestyle='--',label='Mean',linewidth=1.2)
ax.axvline(data.fineagg.median(),color='g',linestyle='--',label='Median',linewidth=1.2)
ax.legend(loc='best')
#histogram plot
plt.subplot(3,3,3)
data.fineagg.plot.hist(color='g').set_title("fineagg Vs Frequency");
Observation¶
- The column is almost Normally Distributed
- Mean and Median are almost the same
- Most of the values are between 731 to 824
- There are outliers
- It has Two Gaussians
age¶
In [18]:
plt.figure(figsize=(20,20))
#boxplot
plt.subplot(3,3,1)
sns.boxplot(data.age,showfliers=True,color='c').set_title("Distribution of age")
#dist plot
ax=plt.subplot(3,3,2)
sns.distplot(data.age,color='m').set_title("age Vs Frequency")
ax.axvline(data.age.mean(),color='r',linestyle='--',label='Mean',linewidth=1.2)
ax.axvline(data.age.median(),color='g',linestyle='--',label='Median',linewidth=1.2)
ax.legend(loc='best')
#histogram plot
plt.subplot(3,3,3)
data.age.plot.hist(color='g').set_title("age Vs Frequency");
Observation¶
- The column is skewed towards right
- Mean and Median are not the same
- Most of the values are between 7 to 56
- There are lot of outliers
- It has Multiple Gaussians
In [19]:
agedata=data.copy()
def age_bin(data):
if data.age <= 30:
return '1 month'
if data.age > 30 and data.age <= 60 :
return '2 months'
if data.age > 60 and data.age <= 90 :
return '3 months'
if data.age > 90 and data.age <= 120 :
return '4 months'
if data.age > 120 and data.age <= 150 :
return '5 months'
if data.age > 150 and data.age <= 180 :
return '6 months'
if data.age > 180 and data.age <= 210 :
return '7 months'
if data.age > 210 and data.age <= 240 :
return '8 months'
if data.age > 240 and data.age <= 270 :
return '9 months'
if data.age > 270 and data.age <= 300 :
return '10 months'
if data.age > 300 and data.age <= 330 :
return '11 months'
if data.age > 330 :
return '12 months'
agedata['age_in_months'] = agedata.apply(lambda data:age_bin(data) , axis=1)
In [20]:
ax=plt.figure(figsize=(10, 6))
sns.countplot(agedata['age_in_months'], order = ['1 month', '2 months', '3 months', '4 months', '6 months', '9 months', '12 months'])
print(agedata['age_in_months'].value_counts())
- From the above plot we found that one month bin has maximum occurence
Target Column (strength) Distribution¶
strength¶
In [21]:
plt.figure(figsize=(20,20))
#boxplot
plt.subplot(3,3,1)
sns.boxplot(data.strength,showfliers=True,color='c').set_title("Distribution of strength")
#dist plot
ax=plt.subplot(3,3,2)
sns.distplot(data.strength,color='m').set_title("strength Vs Frequency")
ax.axvline(data.strength.mean(),color='r',linestyle='--',label='Mean',linewidth=1.2)
ax.axvline(data.strength.median(),color='g',linestyle='--',label='Median',linewidth=1.2)
ax.legend(loc='best')
#histogram plot
plt.subplot(3,3,3)
data.strength.plot.hist(color='g').set_title("strength Vs Frequency");
Observation¶
- The column is skewed towards right
- Mean and Median are not the same
- 25% of the values are 0
- Most of the values are between 0 to 142
- There are outliers
Multivariate Analysis¶
Influence of Different attributes on Strength¶
In [22]:
for col in list(data.columns)[:-1]:
fig,ax1=plt.subplots(figsize=(15,7.5),ncols=1,sharex=False)
sns.regplot(x=data[col],y=data['strength'],ax=ax1).set_title(f'{col} Vs strength')
In [23]:
sns.pairplot(data,diag_kind='kde');
In [24]:
plt.figure(figsize=(25,25))
ax=sns.heatmap(data.corr(),vmax=.8,square=True,fmt='.2f',annot=True,linecolor='white',linewidths=0.01)
plt.title('Correlation of Attributes')
plt.show()
Based on Pair Plot and Correlation Matrix we Understand that¶
Cement does not have any significiant relationship with other Independent attributes But it is positively associated with the targest attribute strength, the relation is not very strong
Slag Does not have any significiant relation with any of the attributes
Ash Does not have any significiant relation with any of the attributes
water Has a Negative Association with superplastic and fineagg and there is no other significant relation with any other attributes
superplastic has Negative association with water and Positive Association with Ash and strength buth this relation is not so strong
courseagg Does not have any significiant relation with any of the attributes
age has a very slight positive association with strength
Outlier Treatment¶
In [25]:
#creating the copy of orginal data set
data1=data.copy()
In [26]:
data1.boxplot(figsize=(40,20));
In [27]:
_, bp = data1.boxplot(return_type='both', figsize=(20,10), rot='vertical')
fliers = [flier.get_ydata() for flier in bp["fliers"]]
boxes = [box.get_ydata() for box in bp["boxes"]]
caps = [cap.get_ydata() for cap in bp['caps']]
whiskers = [whiskers.get_ydata() for whiskers in bp["whiskers"]]
In [28]:
for idx, col in enumerate(data1.columns):
print('Number of outliers in ',col, '-', len(fliers[idx]))
In [29]:
for idx, col in enumerate(data1.columns):
q1 = data1[col].quantile(0.25)
q3 = data1[col].quantile(0.75)
low = q1 - 1.5*(q3 - q1)
high = q3 + 1.5*(q3 - q1)
data1.loc[(data1[col] < low), col] = caps[idx * 2][0]
data1.loc[(data1[col] > high), col] = caps[idx * 2 + 1][0]
In [30]:
# Check the dataset after Outlier treatment
# Check the dataset after Outlier treatment
fig = plt.figure(figsize = (15, 7.2))
ax = sns.boxplot(data = data1.iloc[:, 0:18], orient = 'h')
- All the outliers are treated
Feature Engineering¶
Principal Component Analysis (PCA)¶
In [31]:
#Standardise the data
from scipy.stats import zscore
data_scaled=data1.apply(zscore)
In [32]:
data_scaled
Out[32]:
In [33]:
#Creating a Covariance Matrix
cov_matrix=np.cov(data_scaled.T)
print('Covariance Matrix \n%s',cov_matrix)
In [34]:
#Calculate Eigen Values & Eigen Vectors
e_vals,e_vecs=np.linalg.eig(cov_matrix)
print('Eigen Values \n%s' %e_vals)
print('\nEigen Vectors \n%s'%e_vecs)
In [35]:
# the "cumulative variance explained" analysis
tot=sum(e_vals)
var_exp=[(i/tot)*100 for i in sorted(e_vals,reverse=True)]
cum_var_exp=np.cumsum(var_exp)
print("Cumulative Variance Explained",cum_var_exp)
In [36]:
# Plotting the variance expalained by the principal components and the cumulative variance explained.
plt.figure(figsize=(15,10))
plt.axhline(y=95, color='r', linestyle=':')
plt.bar(range(1, e_vals.size + 1), var_exp, alpha = 0.5, align = 'center', label = 'Individual explained variance')
plt.step(range(1, e_vals.size + 1), cum_var_exp, where='mid', label = 'Cumulative explained variance')
plt.ylabel('Explained Variance Ratio')
plt.xlabel('Principal Components')
plt.legend(loc = 'best')
plt.tight_layout()
plt.show()
Observations:¶
- The above plot shows that, post PCA, employing 5 principal components we are able to explain more than 95% of the variance in the dataset.
In [37]:
# Create a new matrix using the n components
pca=PCA(n_components=6).fit_transform(data_scaled)
In [38]:
#Converting PCA Transformed data from Array to Dataframe to visualise in the pairplot
pca_df=pd.DataFrame(pca)
sns.pairplot(pca_df,diag_kind='kde')
Out[38]:
Observation¶
- From above graphs we found that all the data points are in a cloud distribution
- There is no Relation between any components
- We have to drop 3 Features from the dataset for getting 95% Variance
- We will drop attributes instead of PCA
Decide on complexity of the model, should it be simple linear model in terms of parameters or would a quadratic or higher degree help¶
In [39]:
X=data_scaled.drop(['strength'],axis=1)
y=data_scaled.strength
X.head()
Out[39]:
Splitting of values¶
In [40]:
X_train,X_test,y_train,y_test=train_test_split(X,y,test_size=.30,random_state=1)
In [41]:
print("Shape of Training Data :",X_train.shape)
print("Shape of Testing Data :",X_test.shape)
Linear Regression¶
In [42]:
lr_model=LinearRegression()
lr_model.fit(X_train,y_train)
lr_trainscore=lr_model.score(X_train,y_train)
lr_testscore=lr_model.score(X_test,y_test)
In [43]:
#coefficients
for idx, col_name in enumerate(X_train.columns):
print("The coefficient for {} is {}".format(col_name, lr_model.coef_[idx]))
- From above we found that Superplastic & coarseagg has very weak co-efficients
Regularizing the model¶
- Ridge Model¶
In [44]:
ridge = Ridge(alpha=.3)
ridge.fit(X_train,y_train)
ridge_trainscore=ridge.score(X_train,y_train)
ridge_testscore=ridge.score(X_test,y_test)
print ("Ridge model:", (ridge.coef_))
Lasso Model¶
In [45]:
lasso = Lasso(alpha=0.1)
lasso.fit(X_train,y_train)
lasso_trainscore=lasso.score(X_train,y_train)
lasso_testscore=lasso.score(X_test,y_test)
print ("Lasso model:", (lasso.coef_))
- We found that the coefficients of ash,coarseagg & fineagg is 0 in Lasso Model ,So we can drop these features
In [46]:
score={'Train Score' : {'Regression' : lr_trainscore,
'Ridge':ridge_trainscore,
'Lasso':lasso_trainscore},
'Test Score':{'Regression' : lr_testscore,
'Ridge':ridge_testscore,
'Lasso':lasso_testscore}}
score_df=pd.DataFrame(score)
score_df
Out[46]:
In [325]:
results = pd.DataFrame({'Method':['Regression'], 'Train Accuracy': lr_trainscore,'Test Accuracy':lr_testscore},index={'1'})
tempresultsdf=pd.DataFrame({'Method':['Ridge'], 'Train Accuracy': ridge_trainscore,'Test Accuracy':ridge_testscore},index={'2'})
results=pd.concat([results,tempresultsdf])
tempresultsdf=pd.DataFrame({'Method':['Lasso'], 'Train Accuracy': lasso_trainscore,'Test Accuracy':lasso_testscore},index={'3'})
results=pd.concat([results,tempresultsdf])
results = results[['Method', 'Train Accuracy','Test Accuracy']]
results
Out[325]:
- The accuray scores of Regression and Ridge model is almost the same in Train and test sets and the performance is better compard to Lasso but lasso have acheived the model score by using 3 less features
Checking weather the model complexity increases the model performance or not¶
In [48]:
poly=PolynomialFeatures(degree = 2, interaction_only=True)#quadratic with degree 2
X_poly=poly.fit_transform(X)
Xpoly_train,Xpoly_test,ypoly_train,ypoly_test=train_test_split(X_poly,y,test_size=.30,random_state=1)
In [49]:
Xpoly_train.shape
Out[49]:
In [50]:
lr_model.fit(Xpoly_train,ypoly_train)
print(lr_model.coef_)
In [51]:
#Train accuracy score
lr_model.score(Xpoly_train,ypoly_train)
Out[51]:
In [52]:
#Test accuracy score
lr_model.score(Xpoly_test,ypoly_test)
Out[52]:
In [53]:
ridge.fit(Xpoly_train,ypoly_train)#ridge model
print ("Ridge model:", (ridge.coef_))
In [54]:
lasso.fit(Xpoly_train,ypoly_train)#lasso model
print ("Lasso model:", (lasso.coef_))
In [55]:
print("Ridge Scores: ")
print("train score : ", ridge.score(Xpoly_train,ypoly_train))
print("test score : ", ridge.score(Xpoly_test,ypoly_test))
print()
print("Lasso Scores:")
print("train score : ", lasso.score(Xpoly_train,ypoly_train))
print("test score : ", lasso.score(Xpoly_test,ypoly_test))
Observations¶
- The model Performance is slightly increased
- Quadratic model will be better than a simple linear model
- The low score may be due to the mix of gaussians ,
Explore for gaussians. If data is likely to be a mix of gaussians, explore individual clusters and present your findings in terms of the independent attributes and their suitability to predict strength¶
K Means Clustering¶
Now, we will use K-Means clustering to group data based on their attribute. First, we need to determine the optimal number of groups. For that we conduct the knee test to see where the knee happens.
In [58]:
### Finding Optimal no of clusters
from scipy.spatial.distance import cdist
clusters=range(1,10)
meanDistortions=[]
for k in clusters:
model=KMeans(n_clusters=k)
model.fit(data_scaled)
prediction=model.predict(data_scaled)
meanDistortions.append(sum(np.min(cdist(data_scaled, model.cluster_centers_, 'euclidean'), axis=1)) / data_scaled.shape[0])
plt.plot(clusters, meanDistortions, 'bx-')
plt.xlabel('k')
plt.ylabel('Average distortion')
plt.title('Selecting k with the Elbow Method')
Out[58]:
In [59]:
#Lets try with 3 groups
kmeans_model=KMeans(3)
kmeans_model.fit(data_scaled)
prediction=kmeans_model.predict(data_scaled)
kmeans_df=data_scaled.copy()#creating copy
kmeans_df["Group"]=prediction
kmeans_df.head()
Out[59]:
In [60]:
clusters = kmeans_df.groupby(["Group"])
clusters.mean()
Out[60]:
In [61]:
kmeans_df.boxplot(by='Group',layout=(3,3),figsize=(15,10));
Let us analyze the Strength column vs other columns group wise.¶
In [62]:
#cement vs strength
with sns.axes_style("white"):
plot=sns.lmplot('cement','strength',data=kmeans_df,hue='Group')
plot.set(ylim=(-3,3))
In [63]:
#slag vs cement
with sns.axes_style("white"):
plot=sns.lmplot('slag','strength',data=kmeans_df,hue='Group')
plot.set(ylim=(-3,3))
In [64]:
#ash vs strength
with sns.axes_style("white"):
plot=sns.lmplot('ash','strength',data=kmeans_df,hue='Group')
plot.set(ylim=(-3,3))
In [65]:
#superplastic vs strength
with sns.axes_style("white"):
plot=sns.lmplot('superplastic','strength',data=kmeans_df,hue='Group')
plot.set(ylim=(-3,3))
In [66]:
#coarseagg vs strength
with sns.axes_style("white"):
plot=sns.lmplot('coarseagg','strength',data=kmeans_df,hue='Group')
plot.set(ylim=(-3,3))
In [67]:
#fineagg vs strength
with sns.axes_style("white"):
plot=sns.lmplot('fineagg','strength',data=kmeans_df,hue='Group')
plot.set(ylim=(-3,3))
In [68]:
#water vs strength
with sns.axes_style("white"):
plot=sns.lmplot('water','strength',data=kmeans_df,hue='Group')
plot.set(ylim=(-3,3))
In [69]:
#age vs strength
with sns.axes_style("white"):
plot=sns.lmplot('age','strength',data=kmeans_df,hue='Group')
plot.set(ylim=(-3,3))
Observations from Boxplot and lm plot¶
- The groups are overlapping with eachother
- It is difficult to distinguish the clusters clearly, So it is difficult to pull out a cluster and build a model
- Hence K-means also does not seem to help our cause
- ash, coarseagg and fineagg are week contributors
Decision Tree¶
In [73]:
dt=DecisionTreeRegressor()
dt.fit(X_train,y_train)
Out[73]:
Feature Importance plot using Decision Tree Regressor¶
In [130]:
pd.DataFrame(dt.feature_importances_, index = data_scaled.columns[:-1],
columns=['Importance']).sort_values('Importance',ascending=False).plot(kind='bar',color='c', figsize=(15,7), title='Feature Importance of Decision Tree')
Out[130]:
- cement,age,water and slag are important attributes
- coarseagg,fineagg,superplastic and ash are less important so they will not contribute much to the strength
- we found that coarseagg,fineagg & ash are weak contributors from the correlation map as well as the feature importance in decision tree, Hence we will Drop these columns
In [75]:
#creating copy of the dataset
data_dt=data_scaled.copy()
In [76]:
X=data_dt.drop(['strength','ash','fineagg','coarseagg'],axis=1)
y=data_dt['strength']
# Spliting X&y into training and testing set in the ratio 70:30
X_train,X_test,y_train,y_test=train_test_split(X,y,test_size=.30,random_state=1)
In [77]:
dt2=DecisionTreeRegressor()
dt2.fit(X_train,y_train)
Out[77]:
In [78]:
y_pred=dt2.predict(X_test)
dt_trainacc=dt2.score(X_train,y_train)
dt_testacc=dt2.score(X_test,y_test)
print('Training Accuracy (DT) : ' ,dt_trainacc)
print('Testing Accuract (DT): ',dt_testacc )
- The model is overfitting as the accuracy in train data is 99% and in test it is only 85%
In [79]:
sns.set(style="darkgrid", color_codes=True)
with sns.axes_style("white"):
sns.jointplot(x=y_test, y=y_pred, stat_func=pearsonr,kind="reg", color="k");
In [326]:
tempresultsdf=pd.DataFrame({'Method':['Decision Tree'], 'Train Accuracy': dt_trainacc,'Test Accuracy':dt_testacc},index={'4'})
results=pd.concat([results,tempresultsdf])
results = results[['Method', 'Train Accuracy','Test Accuracy']]
results
Out[326]:
Hyperparameter Tuning with Grid SearchCV¶
In [111]:
param_grid = {'max_depth': np.arange(3, 6),
'criterion' : ['mse','mae'],
'max_leaf_nodes': [100, 105, 90, 95],
'min_samples_split': [6, 7, 8, 9, 10],
'max_features':[2, 3, 4, 5, 6]}
grid_tree_dt = GridSearchCV(DecisionTreeRegressor(), param_grid, cv = 10, scoring= 'r2')
grid_tree_dt.fit(X_train, y_train)
print(grid_tree_dt.best_estimator_)
print('Best Score:', np.abs(grid_tree_dt.best_score_))
In [114]:
# invoking the decision tree classifier function#criterion = 'mse'
dt3 = DecisionTreeRegressor(criterion = 'mae',max_depth=5,min_samples_split=6,max_leaf_nodes=105,max_features=4)
dt3.fit(X_train, y_train)
Out[114]:
In [119]:
y_preddt=dt3.predict(X_test)
dt3_trainacc=dt3.score(X_train,y_train)
dt3_testacc=dt3.score(X_test,y_test)
print('Training Accuracy (DT Hyperparameter tuning) : ' ,dt3_trainacc)
print('Testing Accuracy (DT yperparameter tuning): ',dt3_testacc )
In [120]:
sns.set(style="darkgrid", color_codes=True)
with sns.axes_style("white"):
sns.jointplot(x=y_test, y=y_preddt, stat_func=pearsonr,kind="reg", color="k");
In [327]:
tempresultsdf=pd.DataFrame({'Method':['Decision Tree with Hypeparameter Tuning'], 'Train Accuracy': dt3_trainacc,'Test Accuracy':dt3_testacc},index={'5'})
results=pd.concat([results,tempresultsdf])
results = results[['Method', 'Train Accuracy','Test Accuracy']]
results
Out[327]:
- By Hyper Parameter tuning the over fitting is reduced but ith has brought down model performance as well
Random Forest Regressor¶
In [132]:
#creating copy of the dataset
data_rf=data_scaled.copy()
In [142]:
X=data_rf.drop(['strength'],axis=1)
y=data_rf['strength']
# Spliting X&y into training and testing set in the ratio 70:30
X_train,X_test,y_train,y_test=train_test_split(X,y,test_size=.30,random_state=1)
rf=RandomForestRegressor()
rf.fit(X_train, y_train)
Out[142]:
Feature Importance plot using Random Forest Classifier¶
In [313]:
pd.DataFrame(rf.feature_importances_, index = data_rf.columns[:-1],
columns=['Importance']).sort_values('Importance',ascending=False).plot(kind='bar',color='r', figsize=(15,7), title='Feature Importance of Random Forest')
Out[313]:
- cement,age are important attributes
- coarseagg,fineagg,superplastic and ash are less important so they will not contribute much to the strength
- we found that coarseagg,fineagg & ash are weak contributors from the correlation map as well as the feature importance in decision tree, Hence we will Drop these columns
Dropping of ash, fineagg and coarseagg attributes¶
In [244]:
X=data_rf.drop(['strength','ash','fineagg','coarseagg'],axis=1)
y=data_rf['strength']
# Spliting X&y into training and testing set in the ratio 70:30
X_train,X_test,y_train,y_test=train_test_split(X,y,test_size=.30,random_state=1)
In [245]:
rf2=RandomForestRegressor()
rf2.fit(X_train,y_train)
Out[245]:
In [246]:
y_pred=rf2.predict(X_test)
rf_trainacc=rf2.score(X_train,y_train)
rf_testacc=rf2.score(X_test,y_test)
print('Training Accuracy (RF) : ' ,rf_trainacc)
print('Testing Accuract (RF): ',rf_testacc )
- The model is overfitting as the accuracy in train data is approximately 98% and in test it is only 91%
In [149]:
sns.set(style="darkgrid", color_codes=True)
with sns.axes_style("white"):
sns.jointplot(x=y_test, y=y_pred, stat_func=pearsonr,kind="reg", color="k");
In [328]:
tempresultsdf=pd.DataFrame({'Method':['Random Forest'], 'Train Accuracy': rf_trainacc,'Test Accuracy':rf_testacc},index={'6'})
results=pd.concat([results,tempresultsdf])
results = results[['Method', 'Train Accuracy','Test Accuracy']]
results
Out[328]:
Hyperparameter Tuning with Grid SearchCV¶
In [173]:
param_grid = {'max_depth': np.arange(3, 8),
'criterion' : ['mse','mae'],
'max_leaf_nodes': [100, 105, 90, 95],
'min_samples_split': [6, 7, 8, 9, 10],
'max_features':['auto','sqrt','log2']}
grid_tree_rf = GridSearchCV(RandomForestRegressor(), param_grid, cv = 5, scoring= 'r2')
grid_tree_rf.fit(X_train, y_train)
print(grid_tree_rf.best_estimator_)
print('Best Score:', np.abs(grid_tree_rf.best_score_))
In [186]:
# invoking the decision tree classifier function#criterion = 'mse'
rf3 = RandomForestRegressor(criterion = 'mse',max_leaf_nodes=90,max_depth=7,min_samples_split=6,)
rf3.fit(X_train, y_train)
Out[186]:
In [247]:
y_predrf=rf3.predict(X_test)
rf3_trainacc=rf3.score(X_train,y_train)
rf3_testacc=rf3.score(X_test,y_test)
print('Training Accuracy (RF Hyperparameter tuning) : ' ,rf3_trainacc)
print('Testing Accuracy (RF yperparameter tuning): ',rf3_testacc )
In [192]:
sns.set(style="darkgrid", color_codes=True)
with sns.axes_style("white"):
sns.jointplot(x=y_test, y=y_predrf, stat_func=pearsonr,kind="reg", color="k");
In [329]:
tempresultsdf=pd.DataFrame({'Method':['Random Forest with Hypeparameter Tuning'], 'Train Accuracy': rf3_trainacc,'Test Accuracy':rf3_testacc},index={'7'})
results=pd.concat([results,tempresultsdf])
results = results[['Method', 'Train Accuracy','Test Accuracy']]
results
Out[329]:
Gradient Boosting Regressor¶
In [201]:
#creating copy of the dataset
data_gb=data_scaled.copy()
In [204]:
X=data_gb.drop(['strength'],axis=1)
y=data_gb['strength']
# Spliting X&y into training and testing set in the ratio 70:30
X_train,X_test,y_train,y_test=train_test_split(X,y,test_size=.30,random_state=1)
gb=GradientBoostingRegressor()
gb.fit(X_train, y_train)
Out[204]:
Feature Importance plot using Gradient Boost Regressor¶
In [314]:
pd.DataFrame(gb.feature_importances_, index = data_gb.columns[:-1],
columns=['Importance']).sort_values('Importance',ascending=False).plot(kind='bar',color='g', figsize=(15,7), title='Feature Importance of Gradient Boost Regressor')
Out[314]:
- cement,age are important attributes
- coarseagg,fineagg,superplastic and ash are less important so they will not contribute much to the strength
- we found that coarseagg,fineagg & ash are weak contributors from the correlation map as well as the feature importance in decision tree, Hence we will Drop these columns
Dropping of ash, fineagg and coarseagg attributes¶
In [248]:
X=data_gb.drop(['strength','ash','fineagg','coarseagg'],axis=1)
y=data_gb['strength']
# Spliting X&y into training and testing set in the ratio 70:30
X_train,X_test,y_train,y_test=train_test_split(X,y,test_size=.30,random_state=1)
In [249]:
gb2=GradientBoostingRegressor()
gb2.fit(X_train,y_train)
Out[249]:
In [250]:
y_pred=gb2.predict(X_test)
gb_trainacc=gb2.score(X_train,y_train)
gb_testacc=gb2.score(X_test,y_test)
print('Training Accuracy (GB) : ' ,gb_trainacc)
print('Testing Accuract (GB): ',gb_testacc )
- The model is overfitting as the accuracy in train data is approximately 94% and in test it is only 89%
In [212]:
sns.set(style="darkgrid", color_codes=True)
with sns.axes_style("white"):
sns.jointplot(x=y_test, y=y_pred, stat_func=pearsonr,kind="reg", color="k");
In [330]:
tempresultsdf=pd.DataFrame({'Method':['Gradient Boost'], 'Train Accuracy': gb_trainacc,'Test Accuracy':gb_testacc},index={'8'})
results=pd.concat([results,tempresultsdf])
results = results[['Method', 'Train Accuracy','Test Accuracy']]
results
Out[330]:
Hyperparameter Tuning with Grid SearchCV¶
In [218]:
param_grid = {'n_estimators': [100, 200, 250, 500],
'max_depth': range(10, 31, 2),
'min_samples_split': range(50, 501, 10),
'learning_rate':[0.1, 0.2]}
clf = GridSearchCV(GradientBoostingRegressor(random_state = 1)
, param_grid, cv = 5, scoring= 'r2').fit(X_train, y_train)
print(clf.best_estimator_)
print('Best Score:', clf.best_score_)
In [219]:
# invoking the decision tree classifier function#criterion = 'mse'
gb3 = GradientBoostingRegressor(max_depth=18, min_samples_split=140, n_estimators=500,
random_state=1)
gb3.fit(X_train, y_train)
Out[219]:
In [251]:
y_predgb=gb3.predict(X_test)
gb3_trainacc=gb3.score(X_train,y_train)
gb3_testacc=gb3.score(X_test,y_test)
print('Training Accuracy (GB Hyperparameter tuning) : ' ,gb3_trainacc)
print('Testing Accuracy (GB yperparameter tuning): ',gb3_testacc )
In [221]:
sns.set(style="darkgrid", color_codes=True)
with sns.axes_style("white"):
sns.jointplot(x=y_test, y=y_predgb, stat_func=pearsonr,kind="reg", color="k");
In [331]:
tempresultsdf=pd.DataFrame({'Method':['Gradient Boost with Hypeparameter Tuning'], 'Train Accuracy': gb3_trainacc,'Test Accuracy':gb3_testacc},index={'9'})
results=pd.concat([results,tempresultsdf])
results = results[['Method', 'Train Accuracy','Test Accuracy']]
results
Out[331]:
Ada Boosting Regressor¶
In [223]:
#creating copy of the dataset
data_ab=data_scaled.copy()
In [224]:
X=data_ab.drop(['strength'],axis=1)
y=data_ab['strength']
# Spliting X&y into training and testing set in the ratio 70:30
X_train,X_test,y_train,y_test=train_test_split(X,y,test_size=.30,random_state=1)
ab=AdaBoostRegressor()
ab.fit(X_train, y_train)
Out[224]:
Feature Importance plot using Ada Boost Regressor¶
In [316]:
pd.DataFrame(ab.feature_importances_, index = data_ab.columns[:-1],
columns=['Importance']).sort_values('Importance',ascending=False).plot(kind='bar',color='y', figsize=(15,7), title='Feature Importance of Ada Boost Regressor')
Out[316]:
- cement,age & water are important attributes
- coarseagg,fineagg,superplastic and ash are less important so they will not contribute much to the strength
- we found that coarseagg,fineagg & ash are weak contributors from the correlation map as well as the feature importance in decision tree, Hence we will Drop these columns
Dropping of ash, fineagg and coarseagg attributes¶
In [285]:
X=data_ab.drop(['strength','ash','fineagg','coarseagg'],axis=1)
y=data_ab['strength']
# Spliting X&y into training and testing set in the ratio 70:30
X_train,X_test,y_train,y_test=train_test_split(X,y,test_size=.30,random_state=1)
In [253]:
ab2=AdaBoostRegressor()
ab2.fit(X_train,y_train)
Out[253]:
In [254]:
y_pred=ab2.predict(X_test)
ab_trainacc=ab2.score(X_train,y_train)
ab_testacc=ab2.score(X_test,y_test)
print('Training Accuracy (AB) : ' ,ab_trainacc)
print('Testing Accuract (AB): ',ab_testacc )
In [230]:
sns.set(style="darkgrid", color_codes=True)
with sns.axes_style("white"):
sns.jointplot(x=y_test, y=y_pred, stat_func=pearsonr,kind="reg", color="k");
In [332]:
tempresultsdf=pd.DataFrame({'Method':['Ada Boost'], 'Train Accuracy': ab_trainacc,'Test Accuracy':ab_testacc},index={'10'})
results=pd.concat([results,tempresultsdf])
results = results[['Method', 'Train Accuracy','Test Accuracy']]
results
Out[332]:
Hyperparameter Tuning with Grid SearchCV¶
In [262]:
param_grid = {'n_estimators': [100, 200, 250, 500],
'loss' : ['linear', 'square', 'exponential'],
'learning_rate':[0.1, 0.2]}
clf = GridSearchCV(AdaBoostRegressor(random_state = 1)
, param_grid, cv = 5, scoring= 'r2').fit(X_train, y_train)
print(clf.best_estimator_)
print('Best Score:', clf.best_score_)
In [263]:
# invoking the decision tree classifier function#criterion = 'mse'
ab3 = AdaBoostRegressor(learning_rate=0.2, loss='square', n_estimators=500,
random_state=1)
ab3.fit(X_train, y_train)
Out[263]:
In [264]:
y_predab=ab3.predict(X_test)
ab3_trainacc=ab3.score(X_train,y_train)
ab3_testacc=ab3.score(X_test,y_test)
print('Training Accuracy (AB Hyperparameter tuning) : ' ,ab3_trainacc)
print('Testing Accuracy (AB yperparameter tuning): ',ab3_testacc )
In [266]:
sns.set(style="darkgrid", color_codes=True)
with sns.axes_style("white"):
sns.jointplot(x=y_test, y=y_predab, stat_func=pearsonr,kind="reg", color="k");
In [333]:
tempresultsdf=pd.DataFrame({'Method':['Ada Boost Boost with Hypeparameter Tuning'], 'Train Accuracy': ab3_trainacc,'Test Accuracy':ab3_testacc},index={'11'})
results=pd.concat([results,tempresultsdf])
results = results[['Method', 'Train Accuracy','Test Accuracy']]
results
Out[333]:
Bagging Regressor¶
In [256]:
br=BaggingRegressor()
br.fit(X_train,y_train)
Out[256]:
In [258]:
y_pred=br.predict(X_test)
br_trainacc=br.score(X_train,y_train)
br_testacc=br.score(X_test,y_test)
print('Training Accuracy (BR) : ' ,br_trainacc)
print('Testing Accuract (BR): ',br_testacc )
- The model is overfitting as the accuracy in train data is approximately 98% and in test it is only 90%
In [259]:
sns.set(style="darkgrid", color_codes=True)
with sns.axes_style("white"):
sns.jointplot(x=y_test, y=y_pred, stat_func=pearsonr,kind="reg", color="k");
In [334]:
tempresultsdf=pd.DataFrame({'Method':['Bagging Regressor'], 'Train Accuracy': br_trainacc,'Test Accuracy':br_testacc},index={'12'})
results=pd.concat([results,tempresultsdf])
results = results[['Method', 'Train Accuracy','Test Accuracy']]
results
Out[334]:
Hyperparameter Tuning with Grid SearchCV¶
In [268]:
param_grid = {'n_estimators': [100, 200, 250, 500],
'max_features':[2,4,6,8]}
clf = GridSearchCV(BaggingRegressor(random_state = 1)
, param_grid, cv = 5, scoring= 'r2').fit(X_train, y_train)
print(clf.best_estimator_)
print('Best Score:', clf.best_score_)
In [272]:
# invoking the decision tree classifier function#criterion = 'mse'
br2 = BaggingRegressor(max_features=4, n_estimators=100, random_state=1)
br2.fit(X_train, y_train)
Out[272]:
In [273]:
y_predgb=br2.predict(X_test)
br2_trainacc=br2.score(X_train,y_train)
br2_testacc=br2.score(X_test,y_test)
print('Training Accuracy (BR Hyperparameter tuning) : ' ,br2_trainacc)
print('Testing Accuracy (BR yperparameter tuning): ',br2_testacc )
In [274]:
sns.set(style="darkgrid", color_codes=True)
with sns.axes_style("white"):
sns.jointplot(x=y_test, y=y_predgb, stat_func=pearsonr,kind="reg", color="k");
In [335]:
tempresultsdf=pd.DataFrame({'Method':['Bagging Regressor with Hypeparameter Tuning'], 'Train Accuracy': br2_trainacc,'Test Accuracy':br2_testacc},index={'13'})
results=pd.concat([results,tempresultsdf])
results = results[['Method', 'Train Accuracy','Test Accuracy']]
results
Out[335]:
KNN Regressor¶
In [289]:
error=[]
for i in range(1,30):
knn=KNeighborsRegressor(n_neighbors=i)
knn.fit(X_train,y_train)
pred_i = knn.predict(X_test)
error.append(np.mean(pred_i!=y_test))
In [290]:
plt.figure(figsize=(12,6))
plt.plot(range(1,30),error,color='red', linestyle='dashed',marker='o',markerfacecolor='blue',markersize=10)
plt.title('Error Rate K Value')
plt.xlabel('K Value')
plt.ylabel('Mean error')
Out[290]:
In [292]:
#k=3
knn = KNeighborsRegressor(n_neighbors=3)
knn.fit(X_train, y_train)
Out[292]:
In [295]:
y_predknn=knn.predict(X_test)
knn_trainacc=knn.score(X_train,y_train)
knn_testacc=knn.score(X_test,y_test)
print('Training Accuracy (KNN ) : ' ,knn_trainacc)
print('Testing Accuracy (BKNN ): ',knn_testacc )
In [297]:
tempresultsdf=pd.DataFrame({'Method':['KNN'], 'Train Accuracy': knn_trainacc,'Test Accuracy':knn_testacc},index={'13'})
results=pd.concat([results,tempresultsdf])
results = results[['Method', 'Train Accuracy','Test Accuracy']]
results
Out[297]:
Hyperparameter Tuning with Grid SearchCV¶
In [307]:
param_grid = {'n_neighbors' :range(1, 21, 2),
'weights' :['uniform','distance'],
'metric' : ['euclidean', 'manhattan', 'minkowski']}
clf = GridSearchCV(KNeighborsRegressor()
, param_grid, cv = 5, scoring= 'r2').fit(X_train, y_train)
print(clf.best_estimator_)
print('Best Score:', clf.best_score_)
In [308]:
# invoking the decision tree classifier function#criterion = 'mse'
knn2 = KNeighborsRegressor(metric='euclidean', weights='distance')
knn2.fit(X_train, y_train)
Out[308]:
In [310]:
y_predgb=knn2.predict(X_test)
knn2_trainacc=knn2.score(X_train,y_train)
knn2_testacc=knn2.score(X_test,y_test)
print('Training Accuracy (knn Hyperparameter tuning) : ' ,knn2_trainacc)
print('Testing Accuracy (knn yperparameter tuning): ',knn2_testacc )
In [336]:
tempresultsdf=pd.DataFrame({'Method':['KNN with Hypeparameter Tuning'], 'Train Accuracy': knn2_trainacc,'Test Accuracy':knn2_testacc},index={'14'})
results=pd.concat([results,tempresultsdf])
results = results[['Method', 'Train Accuracy','Test Accuracy']]
results
Out[336]:
In [357]:
results=results.sort_values("Test Accuracy",ascending=False)
fig=plt.figure(figsize=(15,10))
ax=sns.barplot(y="Method", x=("Test Accuracy"),data=results)
total = len(results["Test Accuracy"])
for p in ax.patches:
percentage = '{:.1f}%'.format(100 * p.get_width())
x = p.get_x() + p.get_width() + 0.02
y = p.get_y() + p.get_height()/2
ax.annotate(percentage, (x, y))
plt.show()
Model performance range at 95% confidence level¶
Bootstrap Sampling using Gradient Boosting Regressor¶
In [359]:
data_scaled.head()
Out[359]:
In [360]:
values=data_scaled.values
In [374]:
from sklearn.metrics import r2_score
# Number of bootstrap samples to create
n_iterations = 1000 # Number of bootstrap samples to create
n_size = int(len(data_scaled) * 0.50) # picking only 50 % of the given data in every bootstrap sample
# run bootstrap
stats = list()
for i in range(n_iterations):
# prepare train and test sets
train = resample(values, n_samples=n_size) # Sampling with replacement
test = np.array([x for x in values if x.tolist() not in train.tolist()]) # picking rest of the data not considered in sample
# fit model
model = GradientBoostingRegressor(random_state=1, max_depth=12,
min_samples_split=100, n_estimators=200,
learning_rate=0.2)
model.fit(train[:,:-1], train[:,-1])
# evaluate model
predictions = model.predict(test[:,:-1])
score = r2_score(test[:,-1], predictions) # caution, overall accuracy score can mislead when classes are imbalanced
print(score)
stats.append(score)
In [384]:
# plot scores
pyplot.hist(stats)
pyplot.show()
# confidence intervals
alpha = 0.95 # for 95% confidence
p = ((1.0-alpha)/2.0) * 100 # tail regions on right and left .25 on each side indicated by P value (border)
lower = max(0.0, np.percentile(stats, p))
p = (alpha+((1.0-alpha)/2.0)) * 100
upper = min(1.0, np.percentile(stats, p))
print('%.1f confidence interval %.1f%% and %.1f%%' % (alpha*100, lower*100, upper*100));
Bootstrap Sampling using Random Forest Regressor¶
In [388]:
# Number of bootstrap samples to create
n_iterations = 1000 # Number of bootstrap samples to create
n_size = int(len(data_scaled) * 0.50) # picking only 50 % of the given data in every bootstrap sample
# run bootstrap
stats = list()
for i in range(n_iterations):
# prepare train and test sets
train = resample(values, n_samples=n_size) # Sampling with replacement
test = np.array([x for x in values if x.tolist() not in train.tolist()]) # picking rest of the data not considered in sample
# fit model
model = RandomForestRegressor(n_estimators=100)
model.fit(train[:,:-1], train[:,-1])
# evaluate model
predictions = model.predict(test[:,:-1])
score = r2_score(test[:,-1], predictions) # caution, overall accuracy score can mislead when classes are imbalanced
print(score)
stats.append(score)
In [389]:
# plot scores
pyplot.hist(stats)
pyplot.show()
# confidence intervals
alpha = 0.95 # for 95% confidence
p = ((1.0-alpha)/2.0) * 100 # tail regions on right and left .25 on each side indicated by P value (border)
lower = max(0.0, np.percentile(stats, p))
p = (alpha+((1.0-alpha)/2.0)) * 100
upper = min(1.0, np.percentile(stats, p))
print('%.1f confidence interval %.1f%% and %.1f%%' % (alpha*100, lower*100, upper*100))
Conclusion¶
From above results we found that¶
- Gradient Boosting Regressor and Random Forest Regressor in Performing well on training and testing data compared to other models
- At 95% confidence interval Gradient boosting model gave the best performance between 86.8% and 91.4% which is better than Random forest which is between 82.5 and 88.4%
Feature importance were calculated using decision trees, Random forest regressor, Grandient boost regressor and Ada boost regressor.most of them shown Age & Cement as the Important Feature
Outliers where Identified using boxplot and was replaced by median
- No missing values are present in the model
- Hyper parameter tuning has reduced the Overfitting of models
- K-Means clusturing was not sucessfull as the clusters where overlaping
- PCA is not usefull for models with less attributes
- Complex models will give more Accuracy compared to simple linear models
In [ ]: